feat: cross-browser screenshot baseline storage (RFC 0002)#115
Merged
PAMulligan merged 11 commits intoJul 2, 2026
Merged
Conversation
…storage (RFC 0002) New strict config section selecting the baseline backend (commit | ci-artifact | service), git-vs-LFS storage for the commit backend, the pinned Playwright capture image, and the provenance manifest + policy. Structural checks keep storage=lfs on the commit backend, hold visualBaselines.threshold equal to e2e.crossBrowserDiffThreshold (one cross-engine tolerance), and require browsers to stay within e2e.crossBrowserBrowsers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dual-mode ESM lib implementing the RFC 0002 §7.3 manifest: record rebuilds entries per engine from a capture envelope (sha256, engine, route, breakpoint, host, gitSha; top-level Playwright version + pinned image), sync lets sibling baseline writers refresh entries without touching the envelope, and verify classifies each baseline (ok / untracked / modified / missing-file / foreign-host) plus manifest-level version/image drift and an envelope-match flag that marks local compares as advisory. Drops the inert shebang from lib modules (here and pipeline-config.js): they are always invoked as `node <path>`, and vitest's SSR transform rejects hashbangs in imported modules, which blocked direct lib imports from tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ance verification New cross-browser-baseline CLI (node core + thin bash wrapper) implementing RFC 0002 Phase A for the commit backend: - compare captures current screenshots (or takes --current-dir) and diffs them per-baseline against the committed firefox/webkit/chromium baselines via visual-diff.js at the 0.03 cross-engine threshold, writing a report with per-baseline provenance status - provenance verification runs before diffing: warn policy reports stale/foreign/untracked baselines alongside pixel results; enforce policy excludes flagged baselines from diffing and counts them as failures instead of emitting false pixel diffs - compares from a non-container envelope are marked advisory - blocking stays config-driven (visualBaselines.blocking, default false) with a --blocking override, so Phase A ships non-blocking per the RFC - capture records the manifest envelope; local firefox/webkit captures are permitted but recorded host=local and flagged as foreign (RFC §13 Q4: warn-and-record, never silently trusted) - backends live behind the RFC §8 adapter contract; commit is implemented, ci-artifact/service arrive with Phase C Playwright is resolved dynamically (CBB_PLAYWRIGHT_DIR → cwd → script dir) since it is provisioned per-project, with a clear install hint when absent. Transient cross-browser screenshots/diffs/report are gitignored like the regression equivalents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n it advertises The orchestration cross-browser phase was labeled "Firefox and WebKit screenshot comparison" while its dispatch ran capture-only cross-browser-test.sh (RFC 0002 §2). Phase 7 now dispatches cross-browser-baseline.sh compare (committed same-engine baselines + provenance) across the orchestration dispatch table, the figma workflow skill, visual-qa-verification, and /build-from-figma; the phase declares its shared dev-server resource; cross-browser-test.sh is annotated as an ad-hoc capture utility. Sibling baseline writers stay honest about provenance: regression-test.sh --update-baselines and capture-baselines.sh sync manifest entries for their engines (host=local, no-op without a manifest), and the regression walk is restricted to regressionTesting.browsers via a shared common_find_baselines helper so committed firefox/webkit baselines no longer surface as SKIP noise in the chromium regression report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…baselines capture.mode="container" (the default) now wraps capture in a docker run of the pinned visualBaselines.capture.image: repo mounted at /work, localhost URLs rewritten to host.docker.internal (with host-gateway add-host), and @playwright/test installed at the image-tag version into a scratch dir the CLI resolves via CBB_PLAYWRIGHT_DIR. The container only writes PNGs; provenance is recorded host-side with host=container and the host checkout's gitSha. --dry-run prints the docker command without running it; already-in-container runs (CI `container:` jobs) short-circuit the wrapper; --local remains the untrusted escape hatch; an image tag without a parseable Playwright version is rejected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the determinism gap for CI capture (RFC 0002 Phase B): a prep job reads visualBaselines from pipeline.config.json so the pinned Playwright image stays config-driven, then - PRs compare current screenshots against stored baselines inside that container (provenance-checked; exit code honors visualBaselines.blocking) with diff artifacts and an upserted PR comment - workflow_dispatch recaptures baselines deterministically and opens a reviewable PR with the refreshed PNGs + manifest - pushes to main publish the cross-browser-baselines artifact when the ci-artifact backend is selected (last-green-main source) All jobs guard on baselines/app existing, mirroring the visual-regression job (Aurelius itself carries no app under visual test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setup-baseline-lfs.sh implements the storage="lfs" flag (RFC 0002 §6.1, §13 Q5 → forward-only): idempotently appends the baselines LFS filter to .gitattributes and runs `git lfs install --local` in the CURRENT repo (downstream apps run it in theirs). It refuses unless visualBaselines.storage is "lfs" (--force overrides), supports --dry-run, and prints the destructive `git lfs migrate import` history rewrite as guidance only — never executes it. cross-browser-baseline capture/compare now warn on storage↔.gitattributes drift in both directions (lfs configured without a filter, or a filter left behind on storage=git). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dapter contract Completes the RFC 0002 §8 adapter matrix: - ci-artifact: fetch() locates the last green main run of cross-browser-baselines.yml via gh and downloads the published cross-browser-baselines artifact (baseline root + manifest override); an absent artifact is a clean "no published baselines yet" skip, and a missing gh CLI is a clear BackendError - service: delegated to Chromatic or Percy — compare validates the configured project-token env var, then hands off to the provider CLI (chromatic honors blocking via --exit-zero-on-changes; percy gets a snapshots file generated from the configured routes) compare now consumes the full adapter surface: delegated backends short-circuit the pixel path, and fetch() may redirect the baseline root and manifest to the downloaded artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New docs/regression-testing/cross-browser.md covers the full RFC 0002 flow: determinism rationale, provenance statuses and policies, backend matrix (commit/ci-artifact/service with the §11 exposure-and-cost tradeoffs), LFS adoption (forward-only, migration guidance), CI jobs, the blocking-flip criteria (§13 Q2: ≥20 clean containerized compares), and troubleshooting. Reference sweep: regression-testing README (shared baseline dir + manifest sync note), onboarding architecture/pipeline-configuration tables (visualBaselines key reference), scripts/README, root README script trees, error-recovery Phase 7 row, and CLAUDE.md (script list, Phase 7 diagram line, features bullet, key artifacts, honest script count 40→55). Commits the implementation plan for the branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Caught during an end-to-end smoke against a live app: visual-diff.js's JSON field mismatchPct is a 0-1 RATIO despite the name, and both consumers appended "%" to it — understating diffs 100x (a 7.47% pixel change printed as "0.0747%"). cross-browser-baseline now carries the value as mismatchRatio and formats real percentages in its log lines and report; regression-test.sh converts before printing. Also applies repo Prettier style to the new files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements RFC 0002 — Cross-Browser Screenshot Baseline Storage. Closes #111.
First commit records the RFC's acceptance (approved & merged as PR #112 on 2026-07-01), clearing the implementation gate.
What this delivers, by RFC phase
Phase A — make the signal real (non-blocking)
visualBaselinesconfig section + strict JSON Schema, with structural checks (single cross-engine tolerance, browsers ⊆ E2E matrix, LFS only on the commit backend)baselines/manifest.json): engine, Playwright version, pinned image, sha256, gitSha, capture host —scripts/lib/baseline-manifest.js(record/sync/verify)scripts/cross-browser-baseline.sh|js— capture / compare / verify. Compare diffs firefox/webkit/chromium against committed same-engine baselines viavisual-diff.jsat 0.03cross-browserphase: Phase 7 now runs the comparison it advertises (dispatch table, workflow skills,/build-from-figma);cross-browser-test.shremains as an ad-hoc capture utilityregression-test.shwalks only its own browsers (no SKIP noise from cross-browser baselines) and, likecapture-baselines.sh, syncs manifest entries when it rewrites chromium baselinesPhase B — determinism, then teeth
capture.mode: "container"wraps capture indocker run <pinned image>locally (--dry-runto inspect;--local= untrusted escape hatch), and CI runs compare/capture insidecontainer:jobs with the image read from configpolicy: "warn"reports alongside pixel results;policy: "enforce"excludes flagged baselines and counts them as failures. Non-container compares are marked advisoryblockingconfig +--blocking), but stays off:blocking: falseandqualityGate.crossBrowserScreenshotsRequired: falseare unchanged per the RFC. Flip criteria documented (≥20 clean containerized CI compares)Phase C — the other adapters
storage: "lfs"+scripts/setup-baseline-lfs.sh(idempotent.gitattributesautomation, forward-only; history rewrite printed as guidance only) + drift warnings both directionsci-artifactbackend: compares against thecross-browser-baselinesartifact published from the last greenmain(publish job included); clean skip before the first publishservicebackend: delegated Chromatic/Percy hand-off, token via CI secret, opt-inCI: new
.github/workflows/cross-browser-baselines.yml— PR compare (pinned container, diff artifacts, upserted PR comment),workflow_dispatchdeterministic recapture → baselines PR,mainartifact publish for the ci-artifact backend.Docs:
docs/regression-testing/cross-browser.md(guide incl. §11 exposure/cost table and §13 resolutions), onboarding config/architecture tables, scripts/README, root README, CLAUDE.md, implementation plan.RFC §13 open questions — resolutions
host: "local"→ flagged foreign), never silently trusted; hard-fail only underpolicy: "enforce"git lfs migrate importdocumented, never executedVerification
agent-plugin-lib.test.js, pre-exists on a clean tree on the dev machine — its shebang trips vitest's transform, same class of issue fixed here forpipeline-config.js)--blocking) → tamper detection (modified) → LFS setup in a scratch repovisual-diff.js'smismatchPctis a 0–1 ratio, and both consumers printed it as a percent (understating diffs 100×)docker runcapture (image pull ~2 GB) — the command construction is covered by--dry-runtests, and the in-container path is what CI runs🤖 Generated with Claude Code